{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 自定义层\n",
    "\n",
    "深度学习的一个魅力在于神经网络中各式各样的层，例如全连接层和后面章节中将要介绍的卷积层、池化层与循环层。虽然tf.keras提供了大量常用的层，但有时候我们依然希望自定义层。本节将介绍如何自定义一个层，从而可以被重复调用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.0.0\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = tf.random.uniform((2,20))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.4.1 custom layer without parameters\n",
    "\n",
    "我们先介绍如何定义一个不含模型参数的自定义层。事实上，这和[“模型构造”]一节中介绍的使用`tf.keras.Model`类构造模型类似。下面的`CenteredLayer`类通过继承`tf.keras.layers.Layer`类自定义了一个将输入减掉均值后输出的层，并将层的计算定义在了`call`函数里。这个层里不含模型参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CenteredLayer(tf.keras.layers.Layer):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "\n",
    "    def call(self, inputs):\n",
    "        return inputs - tf.reduce_mean(inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以实例化这个层，然后做前向计算。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=11, shape=(5,), dtype=int32, numpy=array([-2, -1,  0,  1,  2])>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "layer = CenteredLayer()\n",
    "layer(np.array([1,2,3,4,5]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们也可以用它来构造更复杂的模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=42, shape=(2, 20), dtype=float32, numpy=\n",
       "array([[-0.2791378 , -0.80257636, -0.8498672 , -0.8917849 , -0.43128002,\n",
       "         0.2557137 , -0.51745236,  0.31894356,  0.03016172,  0.5299317 ,\n",
       "        -0.094203  , -0.3885942 ,  0.6737736 ,  0.5981153 ,  0.30068082,\n",
       "         0.42632163,  0.3067779 ,  0.07029241,  0.0343143 ,  0.41021633],\n",
       "       [ 0.0257766 , -0.4703896 , -0.9074424 , -1.2818251 ,  0.17860745,\n",
       "         0.11847494, -0.14939149,  0.20248316, -0.140678  ,  0.6033463 ,\n",
       "         0.13899392, -0.08732668,  0.08497022,  0.8094018 ,  0.20579913,\n",
       "         0.40613335,  0.2509889 ,  0.34718364, -0.6298219 ,  0.59436864]],\n",
       "      dtype=float32)>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net = tf.keras.models.Sequential()\n",
    "net.add(tf.keras.layers.Flatten())\n",
    "net.add(tf.keras.layers.Dense(20))\n",
    "net.add(CenteredLayer())\n",
    "\n",
    "Y = net(X)\n",
    "Y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面打印自定义层各个输出的均值。因为均值是浮点数，所以它的值是一个很接近0的数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=44, shape=(), dtype=float32, numpy=-2.9802323e-09>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.reduce_mean(Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.4.2 custom layer with parameters\n",
    "\n",
    "我们还可以自定义含模型参数的自定义层。其中的模型参数可以通过训练学出。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "class myDense(tf.keras.layers.Layer):\n",
    "    def __init__(self, units):\n",
    "        super().__init__()\n",
    "        self.units = units\n",
    "\n",
    "    def build(self, input_shape):     # 这里 input_shape 是第一次运行call()时参数inputs的形状\n",
    "        self.w = self.add_weight(name='w',\n",
    "            shape=[input_shape[-1], self.units], initializer=tf.random_normal_initializer())\n",
    "        self.b = self.add_weight(name='b',\n",
    "            shape=[self.units], initializer=tf.zeros_initializer())\n",
    "\n",
    "    def call(self, inputs):\n",
    "        y_pred = tf.matmul(inputs, self.w) + self.b\n",
    "        return y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面，我们实例化`MyDense`类并访问它的模型参数。我们可以直接使用自定义层做前向计算。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([[ 0.05307531, -0.01968029,  0.00317079],\n",
       "        [-0.03745286, -0.0031012 , -0.0925727 ],\n",
       "        [ 0.00653961, -0.0849395 , -0.00591413],\n",
       "        [-0.03926834,  0.03737333, -0.08176559],\n",
       "        [-0.02961348,  0.00735149, -0.04053285],\n",
       "        [-0.0769348 , -0.01365675,  0.04430145],\n",
       "        [ 0.05790468,  0.06002709,  0.00588025],\n",
       "        [ 0.00912714, -0.04544574, -0.08150417],\n",
       "        [ 0.01794734, -0.06478786, -0.0466853 ],\n",
       "        [ 0.0007794 ,  0.07972597,  0.01827623],\n",
       "        [ 0.04688237,  0.040658  ,  0.04173873],\n",
       "        [ 0.07974287, -0.01226464,  0.03872328],\n",
       "        [ 0.023996  , -0.044014  ,  0.01851312],\n",
       "        [-0.04491149,  0.00450119,  0.03688556],\n",
       "        [ 0.01733875, -0.01641337,  0.06909126],\n",
       "        [-0.07539   , -0.0878872 ,  0.0091918 ],\n",
       "        [-0.00092481, -0.06399333,  0.00150875],\n",
       "        [-0.01826238, -0.06126164, -0.05938709],\n",
       "        [ 0.04794892,  0.03742057, -0.0018529 ],\n",
       "        [ 0.03086024,  0.00513093, -0.04271856]], dtype=float32),\n",
       " array([0., 0., 0.], dtype=float32)]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dense = myDense(3)\n",
    "dense(X)\n",
    "dense.get_weights()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们也可以使用自定义层构造模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=121, shape=(2, 1), dtype=float32, numpy=\n",
       "array([[-0.00446665],\n",
       "       [-0.0158301 ]], dtype=float32)>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net = tf.keras.models.Sequential()\n",
    "net.add(myDense(8))\n",
    "net.add(myDense(1))\n",
    "\n",
    "net(X)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tf2.0",
   "language": "python",
   "name": "tf2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
